Of Fruit Flies and My Summer of Python Packaging
Hello friend!
As with the last newsletter, I'm starting with "it's been a while", however in this case there's been a lot of things to announce!
Hope y'all in the northern hemisphere had a pleasant summer, I managed to see Dublin for EuroPython for the first time and now have opinions about draft Guinness that I'll rather not air in public, because I hope to come back to Ireland some day. At the risk to repeat myself: it was wonderful to see friends live in the flesh again.
Fruit Flies
So, I've published a new blog post entitled How I’m a Productive Programmer With a Memory of a Fruit Fly which, TL;DR, is a love letter to offline API browsers (mainly Dash) combined with an introduction of one of my oldest and least successful FOSS projects doc2dash, and a tutorial how to build your own documentation set bundle and submit it to Dash. Plus one guerrilla paragraph that tries to dispel a few myths about Sphinx, en passant.
I've been planning to write this article in a long time. Partly out of frustration with the lack of doc2dash's success and partly out of frustration of seeing my friends juggle a billion tabs while writing code.
I did expect to be accused of writing an ad (whenever I talk about Dash someone gets angry; this time someone even accused me of being the author of Dash in disguise). What I didn't expect is it to be my most successful blog post on publishing day.
I've been on the Hacker News front page quite a few times already (one of my older articles even made it into the now-defunct Hacker Monthly magazine that I'm still keeping around – that article also jump-started my "career" as a conference speaker, but that's a story for a different day), but this post surpassed the next best by more 2x already:
(interestingly, my currently most successful long-tail article is Waiting in asyncio that didn't have a noticeable initial blip at all – I've been blogging for almost 20 years and I still can't predict what people will like)
I think the reason for the success is simply that it resonates with a lot of people. The overwhelm by necessary information is real and you meet only few people who say of themselves that they have a great memory. Some readers (most likely of only the headline) suggested that it's worth to train your memory like a muscle. That's fair enough, but my point is that at this point there's so much information I need every day, it's like wanting to do a 1,000 lb deadlift. It's humanely possible, but still very unlikely for me to achieve, no matter how much I train.
The second reason is, I think, that people simply appreciate senior developers showing vulnerability. Through a wild coincidence, I've watched Brené Brown's now-seminal TED talk Listening to shame where she asserts that "vulnerability is the birthplace of innovation, creativity and change" on the day of publishing, and that only emboldened me to go on and push the article to the public. Obviously, I've of course got some unsavory comments, but I'm happy to have learned about myself that I simply do not care anymore.
Anyhow! I hope you enjoy it (or enjoyed! this article has brought me the most new subscribers to date, so welcome if that's you!). Interestingly, there's been a bunch of pre-work necessary for me to publish doc2dash 3.0 which led to a bunch of content on a topic of passion and heartbreak: Python packaging.
Hynek's Python Packaging Spectacular
One of my most popular long-tail articles is Sharing Your Labor of Love: PyPI Quick and Dirty – people used to walk up to me at conferences and tell me that they open it whenever they have to publish a Python package. It's from 2013 but thanks to occasional updates it stayed (and remains!) correct.
But now, almost 10 years later, one would expect some progress? And progress we got. I'm happy to report that 99.9% of projects can drop their setup.py files for a nice static pyproject.toml. Please note, that setup.py files are not deprecated as I've heard it asserted in conference talks. What is deprecated is invoking setup.py directly.
Originally, I didn't want to announce the packaging-related content until I've published the last missing piece whose working title is "how to drag your Python package into 2022". It's meant as an add-on to the older one and help people to modernize their projects, because I feel like there's articles on how to start every month, but nothing serious guiding you to improve what you have.
However its mind map currently looks like this:
Sadly, I don't have the time and energy to finish it anytime soon, so let's talk about what I've published so far!
The first one as a proper blog post on Recursive Optional Dependencies in Python and explains how to have optional dependencies (aka extras) with static metadata. I used to take advantage of them being dicts and built them dynamically. Now you can make them refer to each other which is super cool and removes one blocker to go static-only.
Next, as someone who doesn't like Poetry, I was very pleased to learn that my favorite tool for recursively pinning dependencies pip-tools has had support for pyproject.toml files for a while!
In my excitement I ended up writing an TIL pip-tools Supports pyproject.toml and submit a PR to pip-tools, so they advertise it more broadly. I love sharp tools that do one thing well!
The final piece is about a topic that is dear to my heart: distributing Python applications.
Python packaging has a reputation that is worse than the reality, but it is true that it's not user-friendly to make your users deal with Python, virtualenvs and what not. It is also my suspicion, that this is one of the reasons why doc2dash never took off.
Turns out, we've got a solution to that called PyOxidizer which is the first Python app packager I could get to work. For more details see the TIL You Can Build Portable Binaries of Python Applications.
I'm genuinely stoked about this!
Now that's all the prose I've published, however I haven't stopped there. I've also published no less than two new projects that are related to Python packaging.
The first solves the last problem that stopped me from moving from setup.py files: I like to have nice PyPI readmes that are more than static files or static strings. For instance I like to append the changelog for just the new release to it.
After being told multiple times at PyCon US that Hatch is great (it is, and I will definitely blog more about it) and that it has the concept of plugins, I've dived into it and the result is hatch-fancy-pypi-readme, a metadata plugin that allows to assemble your PyPI readme from a motley mix of strings and files (that you can cut and slice into parts) and run regexp-based substitutions over it.
While I got a lot of "huh!?" reactions, some people have seen the value immediately and it looks like that even community heavy-weights like Black or pydantic are gonna use it for their next releases. Check out the growing list in our showcases gallery!
I like to think I've solved a blocking problem here and hope it will help the adoption of static metadata. 🤞
The second project I've published is my first GitHub Action that helps you to verify and debug the packaging of your Python packages. It's called build-and-inspect-python-package (yes, I'm not very creative with the names lately) and does exactly what it says on the tin: it will build your package using standard tools (that means you can use it with any standard-compliant build backend including flit, setuptools, or Poetry), run checks over the result, and extract metadata including the PyPI readme for you to download and inspect:
Given that it's currently used only by one project that is not mine (but still a friend's), not a lot people seem to care about this. But if you do, it's out there! 🤓
Tidbits
TIL
There's only one more TIL that I wrote that hasn't anything to do with packaging and it's about crediting contributors on GitHub.
I'm very big on sharing credit and wondered how to streamline the creation of Co-authored-by: commit trailers. I did figure it out with a little help of GitHub's web API and you can read about it in Easier Crediting of Contributors on GitHub.
FOSS
Most of my FOSS work was about doc2dash 3.0 and the packaging stuff I mentioned above. Therefor, not much else to report. Maybe except that I've started getting more involved with towncrier, a wonderful changelog aggregator that helps preventing merge conflicts in changelogs. I see a lot of potential in the tool (I'm already using it in attrs) and hope I can help pushing it further.
There's also an upcoming attrs release, that's probably gonna be the last one supporting Python 3.6. If that's a problem for you, please let us know.
Fun
Since this tidbits got a bit short, allow me to add a game recommendation: DEATHLOOP is one of the most original and fun games I've played in a long time and it just landed on Xbox and in Microsoft's Game Pass. So if you hadn't the chance to play it yet: strong recommendation from someone who usually doesn't like Rogue-li[kt]e games.
Phew, that's a lot of words! Thanks for hanging on and don't be afraid to hit the reply button!
Best,
—h